Teaching English with Technology, 15(2), 4-18, http://www.tewtjournal.org 


4 


CRITICAL ISSUES IN THE EVALUATION 
OF AN INTERNATIONAL PROJECT DEDICATED TO 
TECHNOLOGY-MEDIATED TBLT (PETALL) 
by Antonio Lopes 

University of Algarve, Portugal 
Campus da Penha, 8005-139 Faro 
alopes @ ualg.pt 


Abstract 

The CEFR encourages teachers to resort to task-based activities in the language classroom. 
However, some resistance has been offered to this approach, due to lack of appropriate 
training and difficulty in meeting some basic practical conditions, as well as in finding 
resources and examples of good practices, as shown in a survey conducted within the scope 
of ETALAGE. PETALL (Pan European Task-based Activities in Language Learning [2013- 
2016]) seeks to construct a transnational strategy for ICT-based task design management and 
aims to design tasks that can be implemented in different educational contexts. This paper 
offers an overview of the internal evaluation procedures to be followed at different levels and 
stages of the project, and discusses the objectives, underlying principles and criteria applied. 
Keywords: Task-based Language Teaching; ICT; European-funded projects; project 
evaluation; CEFR. 


1. Introduction 

The Common European Framework (CEFR) underscores the importance of implementing 
task-based activities in the language classroom by dedicating a whole chapter to the role of 
tasks in language teaching (Chapter 7). Since the CEFR has been used as one of the main 
references in language curriculum design and teacher education across Europe, TBLT has 
also gained ground and has received increasing attention from teachers, syllabus designers 
and researchers. On the other hand, over the past two decades many authors have been 
emphasising the advantages of this approach (Willis & Willis, 2001; Ellis, 2003, 2005; 
Nunan, 2004) and their work has definitively contributed to change not only the teachers’ 
perceptions of TBLT, but also their teaching practices, sensitising them to the need of giving 
their students the opportunity to engage in meaningful communicative interaction covering a 
wide range of situations, topics and activities. 

However, this does not necessarily signify that in practical terms the everyday reality 
of the language classroom in European schools has changed radically in relation to more 
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traditional approaches, which tend to focus more on matters of form and less on the effective 
development of communicative skills, and in which the textbook and workbook continue to 
play a central role in the classroom routines. Even when teachers are willing to give TBLT a 
try, the truth is that contextual, logistical, technical and pedagogical issues not always easy to 
resolve usually stand in the way of success and end up discouraging them. Besides, there has 
also come the realization that TBLT is not without its challenges and pitfalls, especially when 
one runs the risk of playing down crucial aspects of form such as grammar, accuracy and 
complexity by prioritizing fluency and meaning (Shehadeh, 2005; Seyyedi et al., 2013), thus 
jeopardizing a sound development of the learners’ linguistic competence (Yuan, 2001; Ellis & 
Yuan, 2004). 

This shows that advocating TBLT alone—be it through the recommendations made by 
the CEFR, be it through the language syllabi of the different national curricula—is not 
enough. Not in any way less important is to guarantee that the conditions to successfully 
realize its potential are there. One step in that direction is to provide teachers with a space to 
approach TBLT from a more critical point of view, to help them reflect on what works and 
what does not, to enable them to share ideas and experiences, to document insights, to 
encourage collaborative working and to make reliable resources and materials readily 
available to them. 

These are basically the main purposes of PETALL. This acronym stands for Pan 
European Task Activities for Language Learning, an LLP transversal KA2 project, 1 2 funded 
by the European Commission (reference number 530863-LLP-2012-NL-KA2-KA2MP). In 
essence, this project seeks to promote the use of technology-mediated tasks in the language 
classroom by (a) providing examples of good practices that can be easily applied in different 
languages and educational contexts, (b) offering teacher training (TT) courses in ICT-based 

TBLT and (c) setting up an online forum for the exchange of ideas and research, as well as for 

2 

the presentation of new proposals. 

One of the advantages of working in a European project of this nature is that it allows 
teachers and researchers to better understand commonalities and differences across Europe— 
be it in terms of teaching practices, methodological approaches, teacher training policies or 
education systems. In the particular case of ICT-based TBLT, it is even possible to gauge the 


1 LLP stands for Lifelong Learning Programme and KA2 for Key Activity 2: Languages. 

2 At the project website: http://petallproject.wix.com/petall : for more information on the project see Lopes, 


2014. 
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level of receptiveness and/or commitment of policy-makers, institutions and teaching staff to 
educational innovation. 

The consortium responsible for the implementation of the project is composed of ten 
national tandems (from Germany, Greece, Hungary, Italy, the Netherlands, Portugal, Serbia, 
Spain, Turkey and the United Kingdom), each formed by a teacher training department and a 
junior or senior high school. This signifies that one has to deal with a considerable diversity 
of views and competing perspectives to understanding and assessing TBLT (some of them 
more theoretical, others more empirical) which need to be discussed and negotiated before a 
common ground is reached, both within the tandem itself and between tandems. The project 
also entails the participation of experts and other stakeholders from outside the consortium to 
guarantee objectivity and reliability in the process of validation of the project deliverables. 
Therefore, the project has been underpinned by mechanisms of quality assurance and 
monitoring operating at different levels and stages. This paper aims to provide an overview of 
these mechanisms and of the internal evaluation procedures to be followed, as well as to offer 
a critical discussion of the objectives, underlying principles and criteria to be applied. 

2. Unresolved issues in TBLT to be tackled 

Before proceeding with the description of the evaluation process that has been undertaken for 
the project, one must bear in mind that evaluation itself is carried out to ensure, among other 
things, that past difficulties have been overcome and that adequate solutions to previously 
reported problems have been found. 

Some of the partners in the consortium collaborated in a former project also dedicated 
to ICT-based task activities in the language classroom entitled ETALAGE - European Task- 
based Activities in Language Learning: a Good Practices Exchange (reference number 
502162-LLP1-2009-1-NL-Comenius-CMP). A study conducted in the course of this latter 
project (Lopes, 2012) revealed that, in spite of the recommendations of the CEFR and of the 
official syllabi, Portuguese teachers remained reluctant about resorting to TBLT. Their 
answers to a pre-course survey aimed at identifying the critical points in the implementation 
of TBLT in the language classroom fell under the following four broad categories: (1) task 
design and planning; (2) implementation and students’ performance; (3) assessment; (4) 
resources. 

The category that received most responses was the first one. A significant number of 
teachers agreed that it was difficult to find ways to incorporate tasks into classroom routines 
and effectively integrating them into lesson planning, especially in the case of teachers and 
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students who were used to a more traditionally teacher-directed approach. Other hindering 
factors under this category that were reported were the considerable amount of work that the 
design and preparation of a single task sometimes entails, the challenge of coming up with 
suitable strategies to enhance the quality of the student’s learning process, and the problem of 
tailoring tasks that somehow succeed in articulating the learners’ interests and motivations on 
the one hand, and the objectives and contents defined by the official syllabi, on the other. 

The other category that received a significant number of responses was the one 
including implementation and students’ performance. What happens inside the classroom, 
how the activities go, the way in which the learner engages in the tasks, and how the teacher 
manages the implementation of the task are also priority concerns for the teachers. One of the 
questions that received most responses was how to convince learners to engage in the task 
resorting solely to the second or foreign language. One could always argue that this derives 
from the learners’ lack of fluency, which in turn usually results from limited opportunities, 
motivation or confidence to practice speaking, and, therefore, the solution would be to go on 
insisting in task-based activities. However, establishing linear causal chains in this respect is 
far too reductive. Other teaching strategies have also to be considered, especially those that 
take effect in the long run. Another issue that was identified was the teachers’ difficulty in 
simultaneously supervising several groups engaged in the activity and keeping track of their 
progress. In fact, the risk of dispersal of attention and fragmentation of the teacher’s work 
may seriously compromise the success of TBLT. 

The two issues that also scored high in the questionnaire were related to the two last 
categories mentioned above. The first one, which is associated with the first category, was the 
definition of the assessment criteria for the task and the selection of the most appropriate tools 
to collect information about the learners’ performance, either collective or individual. As for 
the second one, the respondents also felt that they lacked or were unable to find relevant 
information about useful resources, in particular tasks tailored to suit their own educational 
context. 

Among other things, this survey showed that a great deal of the problems reported by 
the teachers in the adoption of TBLT derived mostly from the following factors: lack of 
guidance or expertise, especially as far as design and planning are concerned; little or no 
experience in the implementation and management of the activities; and insufficient access to 
resources deemed suitable to their requirements and teaching objectives. Since little 
experience can be ascribed, at least in part, to the other two factors, most solutions would then 
have to revolve around collaboration, planning strategies, and provision of samples of good 
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practices, which should be flexible enough in their design to meet different needs, and 
structured in such a way as to help teachers develop their own proposals. 

The results of this study helped the proponents of PETALL to define the following 
aims for the project: 

1) promote collaborative work between teachers and teacher trainers, both at national and 
international levels, by launching regional networks for the design and implementation 
of ICT-based tasks; 

2) provide access to samples of ICT-based tasks that “travel well”, i.e. tasks that can be 
easily applied regardless of the educational context or of the cultural and geographic 
divides; 

3) support mutual understanding and awareness of linguistic and cultural diversity within 
Europe through ICT-based TBLT; 

4) foster the quality of teacher education and training in technology-mediated TBLT; 

5) encourage the continuous development and update of the teachers’ digital competence. 
The dynamics of transnational collaborative work between teachers in task design and 

management that the consortium is seeking to promote should not only provide answers to 
questions posed by individual teachers, but also provide the conditions to realise cross-border 
collective projects and initiatives involving both teachers and students. Hence the focus on 
tasks that are not tied down to any local contextual factors and that can cut across different 
education systems. 

The consortium is also sensitive to the ways in which evaluation tools are decisive in 
leveraging the quality of teacher education and in enhancing the effectiveness of the language 
learning process. 

3. ICT-based TBLT: a relatively young field of research 

Authors, such as Jane Willis (1996), Rod Ellis (2003), David Nunan (2004) and Dave Willis 
& Jane Willis (2001), have long established the theoretical and methodological framework of 
TBLT and, despite some conceptual differences and further developments, their work remains 
the starting point for any scholarly research in the field. 

One must bear in mind, however, that most of their proposals are centred on activities 
involving face-to-face interaction in traditional classroom settings, and that technology- 
mediated learning processes have not received proper attention as yet. This is partly to be 
explained by the fact that, fifteen to twenty years ago, ICT resources in schools were still 
relatively scarce and teachers were not yet inclined, let alone prepared, to make use of ICT in 
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the foreign language classroom. However, the investment in ICT in education made in Europe 
over the past decade (see European Commission, 2013) has radically changed this scenario 
and has opened a whole new range of possibilities in terms of language learning. The ESSIE 
Survey (SMART 2010/0039) shows that the number of computers for educational purposes 
rose from 9.5 per 100 students in 2006 (ref-area EU25, plus Iceland and Norway) to 15.8 in 
2012 (ref-area European Union, excluding Germany, the Netherlands and the UK), which 
represents an increase of 66.3 percent in a six-year period. In the case of upper secondary 
students, the increase is far more substantial—from 15.8 to 32.6, which means that the figures 
more than doubled (106.3 percent). The number of primary schools (grade 4) having a 
website also rose from 54.8 percent to 71.8 percent in the same period. The increase is not so 
significant in upper secondary schools (by grade 11)—from 85.1 to 92.1 percent. Still, it is 
consistent with an overall trend of steady growth—although the figures can vary significantly 
from country to country. True, there is still much to be done, but, in the face of this statistical 
evidence, the conditions for technology-mediated language learning have improved 
considerably and currently proposals for the implementation of TBLT in the classroom should 
take that fact into account. 

Despite the relative scarcity of references, some authors have already taken some steps 
towards the exploration of the interface between ICT and TBLT. 

Walter Schrooten (2006: 129) summarized the potential of ICT for language learning, 
stating that it not only “allows a high degree of differentiation”, but also “elicits a high degree 
of learner motivation and involvement”, not to mention that it provides “enriched content” 
and “a more intense, multisensory learning process”. As far as the teacher is concerned, 
Schrooten argues that “ICT makes teaching more efficient, since the teacher can focus more 
on supporting learners rather than having to focus on providing content.” Though he was not 
referring directly to TBLT, his arguments can easily be combined with the principles of the 
task-based approach. 

More recently, Thomas & Reinders (2010) edited perhaps the most comprehensive 
volume of studies dedicated to technology-mediated TBLT to date. The two editors intended 
to explore the interface and potential synergies between TBLT and Computer-Assisted 
Language Learning (CALL), now that current technologies provide not only fresh 
opportunities to enhance the language teaching and learning process, but also new 
communication contexts in which language is produced and received. The volume covers a 
considerable number of topics, ranging from intercultural exchanges in TBLT with the 
application of computer-mediated communication (CMC), to TBLT and network-based 
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CALL, teacher education in TBLT and technology, and virtual-world networking 
environments. Due to its scope and content, this pioneering volume has played a central role 
in the definition of the project philosophy. 

4. Project evaluation and quality assurance 

Evaluation plays a central role in assuring the quality and continuous improvement of any 
project, and is of critical importance if such project is in the context of education and funded 
via an external public grant. That is the reason why the consortium agreed from the very 
outset that the general aims of the evaluation process should be the following: 

1) Have a clear-cut image of the quality of the project, of its progress, reach, impact and 
contribution to the teaching and learning process; 

2) Get critical feedback on products, outputs, management and the implementation 
process; 

3) Gain some distance from the process and renew our perspective of the overall design 
of the project; 

4) Detect flaws or weaknesses that need to be addressed in due time; 

5) Find the best strategies to build capacity for improvement; 

6) Validate outcomes; 

7) Get the recognition of peers; 

8) Fine-tune details. 

However, the process of setting up the evaluation model of a project with these characteristics 
and complexity presents its own challenges. There is extensive literature on approaches, 
models and frameworks concerning the evaluation of learning and teaching projects in higher 
education (Huber & Harvey, 2013; Birbeck, 2010; Hashimoto et al. 2010; Chesterton & 
Cummings, 2007; Hedberg et al., 2002). It is neither our intention nor purpose to discuss the 
immense variety of approaches that have been proposed over these past few years. What is 
important to point out is that in the structuration of the evaluation model the consortium had 
to take into account not only the project objectives, outputs, outcomes and the context in 
which it is being developed, but also the roles and positions of the different stakeholders. 

Hence, priority was given to a model that could guarantee, on the one hand, 
objectivity, transparency and comprehensiveness, and, on the other, the engagement of the 
participants and end-users alike. It was also openly assumed that total impartiality was 
virtually impossible to achieve, since some of the actors were going to be either directly 
involved in the project or hired by the management as reviewers. It was thus agreed that 
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disclosure policy would induce a more responsible attitude in the opinions issued. Since the 
reviewers are independent from the partner institutions that constitute the consortium, the 
risks of conflict of interests, rejection of critical evaluations and focus on positive findings 
(Mathison, 1991) have been minimised. 

The consortium also established that the evaluation process should be modelled on the 
principles of participatory evaluation, as those underlying Lawrenz & Huffman’s concept of 
“negotiated evaluation approach” (2003; see also 2002) and DeLuca, Poth & Searle’s study 
(2009), which focuses on the negotiation between the evaluator and the stakeholders as a 
means to secure their commitment and build mutual trust. Equally decisive in setting up the 
architecture of the evaluation process were the postulates of the school-based evaluation 
approach (Ryan, Chandler & Samuels, 2007), according to which evaluation incorporates, on 
the one hand, an external component—in this particular case a supranational authority, the 
European Commission, through the Education, Audiovisual and Culture Executive Agency 
(EACEA)— which provides the normative framework for the project management to measure 
its effectiveness and degree of success in relation to other projects, and, on the other hand, an 
internal component where evaluation is carried out by staff members, evaluators and 
consultants, in accordance with performance standards previously negotiated between the 
partners. 

This means that the project is based on a logic of multi-site evaluation (a number of 
institutions proposing and trialing tasks, and each task in turn being evaluated by end-users 
and an independent reviewer), centrally managed by the project coordination in accordance 
with the reference points set by the consortium, and always considering the evaluation 
benchmarks defined by the EACEA. The overall project is in turn followed by an external 
consultant, who provides critical feedback and input to the project coordination. 

The architecture of the project evaluation can be represented in the following diagram: 
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Figure 1. Diagram of the levels of the evaluation of PETALL. 


The following table shows the bottom-up evaluation process, detailing the roles of the 
different actors at each level of evaluation: 


Table 1. Objects and purposes of the evaluation process at different levels. 


Levels of 

evaluation 

Evaluators 

Object of evaluation 

Purposes 

Form 

EACEA 

Team of Experts 

Project progress and 

final reports 

Checking if the 

objectives were met 

Report 

External 

evaluation 

Educational 

consultant 

Progress of the 

project 

Feedback and advice 

Report; 

participation 

in meetings 

External review 

Academic experts 

in FL teaching 

Tasks and TT courses 

V alidation and 

improvement 

Report 

End-users 

Teacher trainees 

TT courses 

Feedback on impact 

and prospective work 

Questionnaire; 

interview 

End-users 

Teachers 

Tasks trialed and task 

form 

Feedback on 

implementation 

Questionnaire; 

interview 

End-users 

Learners 

Tasks trialed 

Feedback on 

implementation 

Questionnaire 






















Teaching English with Technology, 15(2), 4-18, http://www.tewtiournal.org 


13 


For the operationalization phase, the consortium set out a series of steps to ensure the quality 
and the ongoing evaluation of its main product, i.e. the tasks. Proposals of tasks regarded as 
samples of good practices are first discussed between teaching staff members of the two 
institutions that form the national tandem (the teacher training institution plus the middle or 
secondary school), each contributing with different questions and perspectives (academic, 
practical, technical, etc.) to the design of the task. The proposals are then formulated with the 
aid of a template used by all the members of the consortium to guarantee harmonization of 
procedures and criteria. The template is divided into three parts: 

• the overview, which includes the specification of the language(s), CEFR level, skills 
to be developed, duration of the activity, ICT resources to be deployed and ICT 
competences to be developed; 

• the detailed description of the task, including demonstration of the abilities to be 
acquired, type of product created, product requirements or prerequisites, situation or 
theme, process (the different steps of the task in chronological order), division of 
roles, consolidating activities suggested and success factors or assessment criteria; 

• the didactic added-value of the task and other information, which provides practical 
hints for teachers, additional methodological or didactic comments, reasons why the 
task is proposed as a model of best practices, impact that it is expected to have on the 
teaching practices and attitudes, and finally the reasons why the task is assumed to 
travel well. 

Once this form has been filled in, it is sent to the colleagues of the tandem of one of the two 
neighbouring countries for analysis and proposal of changes and adaptations (if needed), 
before they are trialed and the results of the trialing (which include the reports, evaluation 
questionnaires and products) are in turn examined by an independent expert, who is to issue 
an evaluation report based on the data provided. The results and recommendations in the 
report are then fed back to the proponents of the task, who will introduce the suggested 
changes before the definitive version of the task is published on the project website. 

A somewhat similar procedure has been adopted for the evaluation of the teacher 
training courses. A template was also produced for the teacher training institutions to draw up 
their proposals. It should be noted that, in some countries, in-service teacher training courses 
can only be delivered after accreditation by an independent government body. Therefore, the 
template is not to be regarded as the end-product itself, but rather as a guide to the topics that 
are usually covered in this type of training. The template comprises four main sections: 
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1. Scientific background, which includes the rationale and justification for the course, 
investigation of the field (state of the art), innovative outreach, and bibliography and 
other sources; 

2. Course syllabus, which specifies objectives, contents, methodologies, assessment 
procedures, schedule, workload, and expected outcomes (changes in teaching 
practices, adoption of new procedures, production of materials, etc.); 

3. Follow-up strategy, including exploitation of the materials produced, collection of 
evidence of post-course impact, constitution of groups of teachers for future 
collaboration in the project, etc.; 

4. Identification of the accrediting body and other details on the accreditation process. 
Since accreditation by the government body already entails the validation of the teacher 
training course, the evaluation by the independent expert—again, based on the data from end- 
users—is there to provide constructive critical feedback to improve the final version of the 
course to be published on the project website. 

5. Criteria applied 

The fact that different stakeholders perceive project outcomes subjectively constructing their 
own representations and evaluation criteria (McLeod et al., 2012) renders the evaluation 
process problematic, all the more so if the differences that separate them hinge on cultural 
factors. It is therefore important to bridge such divergent views through an initial dialogue 
between the members of the consortium to establish the ground rules for the selection of 
meaningful and commonly understood criteria. 

In the case presented, each evaluator was assigned a separate set of criteria meant to 
address the specific requirements of the object of evaluation. For the final evaluation report, 
the expert evaluators follow the roadmap previously outlined by the EACEA. The two main 
parameters they use, namely effectiveness and coherence, were defined as the starting point of 
the remaining criteria. 

Thus, the criteria set for each evaluation level are as follows, 
i) The final evaluation of the project by EACEA experts 

(1) Objectives met? Results and products delivered? 

(2) Coherence between work programme and activities? 

(3) Effective partnership ? 

(4) Sound management ? 

(5) Sound financial management ? 
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(6) Evaluation and/or quality assurance implemented? 

(7) Effective dissemination ? 

ii) The ongoing evaluation of the project by the external evaluator 

(1) Effectiveness: How far have the objectives of intervention been achieved? Are 
the effects produced the ones that were expected? 

(2) Efficiency: How well are the resources/inputs converted into 
outcomes/products/results? 

(3) Impact: What kind of effects (primary, secondary, short-term, long-term, 
direct, indirect) are produced by the project? How is that reflected in the 
responses of the target groups? 

(4) Coherence: How coherent are the activities and outcomes taking into account 
the objectives to be achieved? 

(5) Relevance: How consistent are the project objectives and outcomes with the 
requirements and needs of the target groups? 

iii) The evaluation of the national courses by independent reviewers and teacher 

trainees 

(1) Overall quality of design 

(2) Coherence: Does the course form a coherent whole? 

(3) Clarity of instructions and presentation of course: Is it clear what trainees are 
supposed to do? 

(4) Effectiveness: Does the course achieve the stated aims? 

(5) Up-to-dateness: Does course contain up-to-date information? 

(6) Appropriateness for target group: Does the course address the needs of the 
target group? 

iv) The evaluation of the tasks by independent reviewers and teachers: 

(1) Formatting: Is the learning task in the prescribed format? 

(2) Level and adequacy : Is the learning task at the correct level of the Common 
European Framework? Does it adequately respond to the needs of students? 

(3) Product /product requirements / situation / theme: The description of the task 
and the specifications of the final product. 

(4) Detailed planning programme: What is the quality of the working procedures 
to help students do the task? 

(5) Resources: Are there sources of information available and to what extent are 
they functional for the execution of the task? 
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(6) Assessment The criteria for the final product on the basis of which students 
will be assessed. To what extent do they provide clarity for students to plan 
ahead and do their job, including their share in the realisation of a collective 
outcome/result, if applicable? 
v) The evaluation of the tasks by the learners 

(1) Relevance'. This activity taught me a lot of things/few things 

(2) Adequacy. The activity was easy/difficult 

(3) Engagement : I had a lot of fun/no fun 

(4) Practical usefulness : The activity helped me to improve my [FL/SL] 

(5) Incidence ofICT in LL: I learned how to use technological resource(s) A (and 
B) to communicate and/or express my ideas 

6. Final remarks 

There are all sorts of tasks that can be proposed in the language classroom and it is the 
teacher’s job to ensure that the activities that are being proposed to the learners draw on their 
interests and trigger their motivation. Resorting to ICT in TBLT can be the right means to 
attract attention of our “digital natives” and increase their progress in lessons. ICT-based tasks 
can be undertaken in contexts (work-related or social) where technology plays an important 
part in conditioning the uses that we make of language, be it in terms of relating experiences, 
establishing inter-personal relationships, or organizing information (Eggins, 2004). A project 
aiming to promote this type of approach must help teachers perceive its potential, monitor and 
manage risks, and guide them through the design and implementation processes. That cannot 
be achieved without a credible evaluation strategy. 

The eligibility period of PETALL ends in June 2016. Currently, tasks are still being 
trialed in the neighbouring countries and the independent reviewers are now in the process of 
being contacted and hired. It is therefore too soon to determine whether the evaluation model 
that has been set up for this project is actually going to yield the expected benefits. 
Nonetheless, although practical issues may need to be resolved in the future, the groundwork 
has been done and the structure is now in place. The project coordination is monitoring the 
evaluation process and seeing to it that partners and stakeholders assume their responsibilities 
in this particular aspect of the project, which constitutes an essential component of the quality 
assurance system. 
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